• 专利附录
  • 专利说明
  • 权利要求
  • 类似技术
  • 同族专利
  • 引用文献
  • 法律状态
  • 优先权
    • 专利摘要:
    SYSTEM AND METHOD FOR GENERATING A GROSS DEPTH MAP. A system and method for generating raw depth maps includes a depth estimation device that creates a depth map pyramid structure that includes a plurality of depth map levels that each have different resolution characteristics. In one embodiment, the depth map levels include a fine-scale depth map, a medium-scale depth map, and a coarse-scale depth map. The depth estimation device evaluates depth values from the fine-scale depth map using fine-scale confidence characteristics, and evaluates depth values from the medium-scale depth map and the coarse-scale depth map using coarse scale confidence characteristics. The depth estimation device then merges optimal depth values from the different depth map levels into an optimal depth map.
    公开号:BR102012030034B1
    申请号:R102012030034-6
    申请日:2012-11-26
    公开日:2021-05-25
    发明作者:Gazi Ali；Pingshan Li；Akira Matsui；Takami Mizukura
    申请人:Sony Corporation；
    IPC主号:
    专利说明:

    FUNDAMENTALS SECTION 1. Field of Invention
    [001] This invention relates generally to techniques for analyzing image data, and more particularly relates to a system and method for using scene detection in a depth estimation procedure. 2. Description of the Basic Technique
    [002] Implementing efficient methods to analyze image data is a significant consideration for contemporary electronic device designers and manufacturers. However, efficiently analyzing image data with electronic devices can create substantial challenges for system designers. For example, improved demands for increased device functionality and performance may require more system processing power and require additional hardware features. An increase in processing or hardware requirements can also result in a corresponding detrimental economic impact due to increased production costs and operational inefficiencies.
    [003] In addition, enhanced device capability to perform various advanced operations can provide additional benefits to a system user, but can also place increased demands on the control and management of various device components. For example, an improved electronic device that effectively analyzes digital image data can benefit from an effective implementation because of the plurality and complexity of the digital data involved.
    [004] Due to growing demands on system characteristics and substantially increasing data magnitudes, it is apparent that development of new techniques for analyzing image data is a matter of concern for related electronic technologies. Therefore, for all of the above reasons, developing effective systems for analyzing image data remains a significant consideration for designers, manufacturers, and users of contemporary electronic devices. SUMMARY
    [005] According to the present invention, a system and method to generate raw depth maps using a multiple resolution procedure are described. In one embodiment, a depth estimation device initially generates a level 2 depth map using any effective techniques. For example, the level 2 depth map can be implemented with a compression ratio that produces a relatively fine-scale resolution. The depth estimator calculates raw confidence values corresponding to the respective depth values from the level 2 depth map using any effective techniques. The depth estimation device then categorizes the depth values according to their respective confidence values into high confidence depth values, medium confidence depth values, and low confidence depth values.
    [006] The depth estimation device also generates a level 1 depth map and a level 0 depth map using any effective techniques. For example, the level 1 depth map can be implemented with a compression ratio that produces a medium scale resolution, and the level 0 depth map can be implemented with a compression ratio that produces a relatively coarse scale resolution. .
    [007] The depth estimating device calculates raw confidence values corresponding to the respective depth values of the level 1 depth map and the level 0 depth map using any effective techniques. The depth estimation device then uses the calculated confidence values to identify any reliable depth value candidates from the level 0 depth map and the level 1 depth map according to predetermined reliability criteria.
    [008] The following depth estimation device performs a scaling procedure on the blocks from the level 1 depth map and the level 0 depth map to fit with the block size of the level 2 depth map using any effective techniques. The depth estimator also performs a magnitude scaling procedure to scale the depth values from the level 1 depth map and the level 0 depth map to fit with the depth value range of the depth map level 2 using any effective techniques.
    [009] The depth estimating device then determines whether any additional reliable depth value candidates have been identified from the level 1 depth map or the level 0 depth map. The depth estimating device marks any depth values unreliable as outliers that are inadequate to populate the final optimized depth map. The depth estimation device also uses any reliable depth value candidates to update the optimal depth values to the final optimized depth map according to any effective techniques. For example, a depth value with the optimal confidence measure can be selected, or a weighted or unweighted average calculation method can be used to combine several different reliable depth values.
    [0010] The depth estimation device advantageously combines the optimal depth values from the depth maps of different levels to generate the final optimized depth map. Finally, the depth estimation device can create a confidence map based on confidence values corresponding to the optimal depth values of the final optimized depth map. The process can then end. The present invention, therefore, provides an improved system and method for generating a depth map using a multiple resolution procedure. BRIEF DESCRIPTION OF THE DRAWINGS
    [0011] FIG. 1 is a block diagram for an embodiment of a camera device in accordance with the present invention; FIG. 2 is a block diagram for an embodiment of the image capture system of Fig. 1 in accordance with the present invention; FIG. 3 is a block diagram for an embodiment of the control module of Fig. 1 in accordance with the present invention; FIG. 4 is a block diagram for an embodiment of the memory of Fig. 3 in accordance with the present invention; FIG. 5A is a diagram for an embodiment of a simplified depth map in accordance with the present invention; FIG. 5B is a diagram of an exemplary embodiment for capturing an out-of-focus blurred image; FIG. 5C is a graph of one modality of an exemplary fit curve; FIG. 6 is a diagram of a multi-resolution pyramid structure, in accordance with an embodiment of the present invention; FIGS. 7A-7C are a flowchart of method steps for generating a depth map using a multiple resolution procedure, in accordance with an embodiment of the present invention; FIG. 8A is a fine-scale diagram of confidence characteristics in accordance with an embodiment of the present invention; FIG. 8B is a diagram of coarse-scale confidence characteristics, in accordance with an embodiment of the present invention; FIG. 8C is a graph illustrating a SAD convergence technique, in accordance with an embodiment of the present invention; and FIG. 9 is a diagram of a simplified confidence map in accordance with an embodiment of the present invention. DETAILED DESCRIPTION
    [0012] The present invention relates to an improvement in image data analysis techniques. The following description is presented to enable someone of simple skill in the art to make and use invention and is provided in the context of a patent application and its requirements. Various modifications to the modalities described will be readily apparent to those skilled in the art, and the generic principles herein may be applied to other modalities. Therefore, the present invention is not intended to be limited to the embodiments shown, but is to be accorded the broadest scope consistent with the principles and features described herein.
    [0013] The present invention comprises a system and method for generating raw depth maps, and includes a depth estimation device that creates a depth map pyramid structure that includes a plurality of depth map levels each of which has different resolution characteristics. In one embodiment, the depth map levels include a fine-scale depth map, a medium-scale depth map, and a coarse-scale depth map. The depth estimation device evaluates depth values from the fine-scale depth map using fine-scale confidence characteristics, and evaluates depth values from the medium-scale depth map and the coarse-scale depth map using coarse scale confidence characteristic. The depth estimation device then assembles optimal depth values from the different depth map levels into an optimal depth map.
    [0014] Referring now to Fig. 1, a block diagram for one embodiment of a camera device 110 is shown, in accordance with the present invention.
    [0015] In the embodiment of FIG. 1, camera device 110 may include, but is not limited to, an image capture subsystem 114, a system bus 116, and a control module 118. In the embodiment of FIG. 1, image capture subsystem 114 can be optically coupled to a photographic target 112, and can also be electrically coupled via system bus 116 to control module 118.
    [0016] In alternative embodiments, camera device 110 may readily include various other components in addition to, or in place of, those components discussed in conjunction with the embodiment of FIG. 1. In addition, in certain embodiments, the present invention may alternatively be incorporated into any suitable type of electronic device other than camera device 110 of Fig. 1. For example, camera device 110 may alternatively be implemented as a device processing equipment, a computer device, or a consumer electronics device.
    [0017] In the embodiment of FIG. 1, since image capture subsystem 114 of camera 110 is automatically focused on target 112, a user of the camera can request camera device 110 to capture image data corresponding to target 112. Control module 118 can then preferably instruct image capture subsystem 114 via system bus 116 to capture image data representing target 112. The captured image data can then be transferred over system bus 116 to control module 118, which can responsively perform various processes and functions with the image data. System bus 116 may also bidirectionally pass various state and control signals between image capture subsystem 114 and control module 118.
    [0018] Referring now to Fig. 2, a block diagram for an embodiment of the image capture subsystem 114 of Fig. 1 is shown, in accordance with the present invention. In the embodiment of FIG. 2, image capture subsystem 114 preferably comprises, but is not limited to, a shutter 218, a lens 220, an image sensor 224, red, green, and blue (R/G/B) amplifiers 228, a analog to digital (A/D) converter 230, and an interface 232. In alternative embodiments, image capture subsystem 114 may readily include various other components in addition to, or in place of, those components discussed in conjunction with the embodiment of FIG. two.
    [0019] In the embodiment of FIG. 2, image capture subsystem 114 can capture image data corresponding to target 112 via reflected light impacting image sensor 224 along optical path 236. Image sensor 224, which may preferably include a charge-coupled device ( CCD), can responsively generate a set of image data representing the target 112. The image data can then be routed through amplifiers 228, A/D converter 230, and interface 232. From interface 232, the image data passes over system bus 116 to control module 118 for proper processing and storage. Other types of image capture sensors such as CMOS or linear arrays are also contemplated to capture image data in conjunction with the present invention. The use and functionality of camera 110 is further discussed below in conjunction with FIGS. 3 - 9.
    [0020] Referring now to Fig. 3, a block diagram for an embodiment of the control module 118 of Fig. 1 is shown, in accordance with the present invention. In the embodiment of FIG. 3, control module 118 preferably includes, but is not limited to, a display 308, a Central Processing unit (CPU) 344, a memory 346, and one or more input/output(s) (I/O) interface. ) 348. Display 308, CPU 344, memory 346, and I/O interface 348 preferably are each coupled to and communicate via the system bus with a subtitle production unit 116 which also communicates with image capture subsystem 114. In alternative embodiments, control module 118 may readily include various other components in addition to, or in place of, those components discussed in conjunction with the embodiment of FIG. 3.
    [0021] In the embodiment of FIG. 3, CPU 344 can be implemented to include any suitable microprocessor device. Alternatively, CPU 344 can be implemented using any other appropriate technology. For example, CPU 344 can be implemented to include certain application-specific integrated circuits (ASICs) or other appropriate electronic devices. Memory 346 can be implemented as one or more suitable storage devices, including, but not limited to, read-only memory, random access memory, and various types of non-volatile memory, such as floppy disk devices, hard disk devices. , or flash memory. I/O interface 348 can provide one or more interfaces effective to facilitate bidirectional communications between camera device 110 and any external entity, including a system user or other electronic device. The 348 I/O interface can be implemented using any appropriate input and/or output devices. The operation and use of control module 118 is further discussed below in conjunction with FIGS. 4 to 9.
    Referring now to Fig. 4, a block diagram for a memory embodiment 346 of Fig. 3 is shown, in accordance with the present invention. In the embodiment of FIG. 4, memory 346 may include, but is not limited to, a camera application 412, an operating system 414, a depth estimation device 416, image data 418, estimate data 420, depth values 422, and values 424. In alternative embodiments, memory 346 may include various other components in addition to, or in place of, those components discussed in conjunction with the embodiment of FIG. 4.
    [0023] In the embodiment of FIG. 4, the camera application 412 may include program instructions that are preferably executed by the CPU 344 (FIG. 3) to perform various functions and operations for the camera device 110. The particular nature and functionality of the camera application 412 preferably varies depending on factors such as the type and particular use of the corresponding camera device 110,
    [0024] In the embodiment of FIG. 4, operating system 414 preferably controls and coordinates low-level functionality of camera device 110. In accordance with the present invention, depth estimation device 416 can control and coordinate a depth estimation procedure to generate depth maps in the camera 110. In the embodiment of FIG. 4 , image data 418 may include one or more images of a photographic target 112 captured by the camera device transmission data generation unit 110. Estimate data 420 may include any types of information or data to perform a data generation procedure. depth map.
    [0025] In the embodiment of FIG. 4, depth values 422 may represent distances between camera 110 and various portions of a photographic target or scene. Depth values 422 can be used to populate a depth map, as discussed below in conjunction with FIG. 5A. Confidence values 424 represent relative probabilities that the respective depth values 422 are corrected. Additional details regarding an operation of depth estimation device 416 are further discussed below in conjunction with FIGS. 5 - 9.
    [0026] Referring now to Fig. 5A, a diagram for one embodiment of a simplified depth map 522 is shown, in accordance with the present invention. Depth map 522 in Fig. 5A is presented for illustrative purposes, and in alternative embodiments, depth maps may include various other components and configurations in addition to, or in place of, those components and configurations discussed in conjunction with the embodiment. of FIG. 5A. For example, for purposes of simplicity, the depth map 522 of Fig. 5A is shown with only four depth values. However, depth maps with any number of depth values are also contemplated in the present invention.
    [0027] In the embodiment of FIG. 5A, depth map 522 corresponds to a particular image data set in which individual depth values represent distances between camera 110 (FIG. 1) and various portions of the photographic target or scene. In certain embodiments, initial depth values for an initial depth map can be determined directly from raw image data in any effective way. In the example of FIG. 5A, depth map 522 includes an A depth value 526(a), a B depth value 526(b), a C depth value 526(c), and a D depth value 526(d). Additional details regarding depth map generation are further discussed below in conjunction with FIGS. 6 - 9.
    [0028] Referring now to Fig. 5B, a diagram of an exemplary embodiment for capturing an out-of-focus blurred image 518 is shown, in accordance with the present invention. The embodiment of FIG. 5B is provided for purposes of illustration, and in alternative embodiments, the present invention may utilize various other configurations and elements for blurry out-of-focus images 518.
    [0029] In the embodiment of FIG. 5B, a sensor 224 of a camera 110 (see FIG. 2) can capture an out-of-focus blurred image 518 of a photographic target or scene 112 to perform a depth estimation procedure. Blurred out-of-focus image 518 can be created by adjusting lens 220 to a position other than the position of the correct focusing lens which depends on the relative positions of target 112, lens 220, and sensor 224.
    [0030] In one embodiment, two different out-of-focus blurred images 518 can be compared to derive a depth estimate. A blur difference can be calculated for two blurred images 518 that are, for example, a depth of field away from each other. A slope of a known fit curve and blur difference can be used to determine the depth of a particular target 112.
    [0031] Referring now to Fig. 5C, a graph of an exemplary fit curve 714 is shown, in accordance with an embodiment of the present invention. The embodiment of FIG. 5C is presented for purposes of illustration, and in alternative embodiments, the present invention may be implemented to use fit curves with settings and parameters in addition to, or in place of, certain of those settings and parameters discussed in conjunction with the embodiment of FIG. . 5C.
    [0032] In certain modalities, a blurred image 1 and a blurred image 2 more out of focus can be captured, the sharper image 1 can be convolved with a Gaussian function (eg a 3 x 3 Gaussian matrix with small variance ) to produce a 1 involved image. Convolved image 1 is compared with blurred image 2. This process is repeated until the two blurred images fit together. The number of iterations can then be plotted against depth of field (or image numbers in increments of a DOF) to produce a blur-fit curve that can be used to estimate the distance from any out-of-focus position to position. in focus. Additional details regarding the prior depth estimation technique are further discussed in US Patent No. 8,045,046 to Li et al., which is incorporated herein by reference.
    [0033] Referring now to Fig. 6, a diagram of a multi-resolution pyramid structure 614 is shown, in accordance with an embodiment of the present invention. The multi-resolution pyramid structure 614 of FIG. 6 presented for purposes of illustration, and in alternative embodiments, the multi-resolution pyramid structure 614 may include various other components and configurations in addition to, or in place of, those components and configurations discussed in conjunction with the embodiment of FIG. 6.
    [0034] In the embodiment of FIG. 6, the multi-resolution pyramid structure 614 includes a level 2,618 depth map, a 1,622 level depth map, and a 0628 level depth map each of which has a different resolution level. In other embodiments, any other effective number or setting of resolution levels may be similarly used. In the embodiment of FIG. 6, all resolution levels 618, 622, and 628 are compressed from the same image data setting, but each level has a different compression ratio. For example, in certain embodiments, level 0 can have a compression ratio of 1/16, level 1 can have a compression ratio of 1/8, and level 2 can have a compression ratio of 1/4. Compression ratios can be achieved in any effective way. For example, a subsampling compression technique can be used.
    [0035] In the embodiment of FIG. 6, each level 618, 622, and 628 includes one or more blocks. For example, level 0 has a single 632 block, level 1 has four blocks arranged in a 2 x 2 matrix, and level 2 has sixteen blocks arranged in a 4 x 4 matrix. number of pixels. For example, in certain modalities, each block can represent 16 x 16 depth values. Due to the different compression ratios, the three different levels 628, 622, and 618 each represent the same area as the original image data. For example, the single level 0 block 632 corresponding to level 1 blocks 636, 640, 644, and 648. Similarly, each level 1 block corresponds to four level 2 blocks. For example, the level 636 block 1 corresponds to level 2 blocks 652, 656, 660, and 664.
    [0036] In the embodiment of FIG. 6, each block of all three levels has an associated depth value. Therefore, level 0 has a depth value, level 1 has four depth values, and level 2 has sixteen depth values. In certain alternative embodiments, the three levels may be implemented to include depth/block values that differ in quantity from the particular configuration shown in the embodiment of FIG. 6. The present invention advantageously evaluates these various depth values with raw confidence measurement techniques, and then joins the optimal depth values to generate a final depth map. Additional details regarding the use of multi-resolution pyramids are further discussed below in conjunction with FIGS. 7 - 9.
    [0037] Referring now to Figs. 7A-7C, a flowchart of method steps for generating a depth map using a multiple resolution procedure is shown, in accordance with an embodiment of the present invention. The embodiment of FIG. 7 is presented for purposes of illustration, and in alternative embodiments, the present invention may readily utilize various steps and sequences other than those steps and sequences discussed in conjunction with the embodiment of FIG. 7.
    [0038] In the embodiment of FIG. 7A, at step 714, a depth estimating device 416 (FIG. 4) or other appropriate entity generates a level depth map 2618 using any effective techniques. For example, the 2618 level depth map can be implemented with a compression ratio that produces a relatively fine-scale resolution. At step 718, depth estimating device 416 can then calculate raw confidence values corresponding to respective depth values from level 2618 depth map using any effective techniques. At step 722, the depth estimation device 416 categorizes the depth values according to their respective confidence values into high confidence depth values 734, medium confidence depth values 730, and low confidence depth values 726. The process of FIG. 7A then advances to Fig. 7C through connecting letters “A” and “B.”
    [0039] In the embodiment of FIG. 7B, at step 738, depth estimating device 416 or other appropriate entity generates a level depth map 1622 and a level depth map 0628 using any effective techniques. For example, level depth map 1,622 can be implemented with a compression ratio that produces a medium-scale resolution, and level depth map 0628 can be implemented with a compression ratio that yields a relatively low resolution. coarse scale.
    [0040] In step 742, the depth estimating device 416 can then calculate raw confidence values corresponding to the respective depth values from the level depth map 1622 and the level depth map 0628 using any effective techniques . In step 750, the depth estimation device 416 uses the calculated confidence values to identify any reliable depth value candidates from the 0626 level depth map and the 1,622 level depth map according to predetermined criteria. of reliability.
    [0041] In step 750, the depth estimating device 416 performs a scaling procedure on the level 1 622 depth map and 0628 level depth map blocks to fit with the block size of the depth map of level 2 618 using any effective techniques. For example, various spatial interpolation techniques such as the bicubic or spline method can be used. In certain embodiments, an edge-preserving interpolation algorithm can be used to perform scaling in which a 2X interpolation core-based mask is used in conjunction with a filter matrix. An example of edge preservation is shown below.

    [0042] In step 754, the depth estimating device 416 performs a magnitude scaling procedure to scale the depth values of the level 1 622 depth map and the 0628 level depth map to fit with the range of depth value of level 2 618 depth map using any effective techniques. For example, the design procedure can use a theoretical approach or a data-based approach. The process in FIG. 7B can then proceed to step 758 of Fig. 7C via the connecting letter "A."
    [0043] In step 758, the depth estimating device 416 determines whether any additional reliable depth value candidates have been identified from the level depth map 1622 or the level depth map 0628. In step 762, depth estimation device 416 marks any unreliable depth values as outliers that are inadequate to populate the final optimized depth map. At step 766, depth estimation device 416 uses any reliable depth value candidates to update the optimal depth values to the final optimized depth map according to any effective techniques. For example, a depth value with the optimal confidence measure can be selected, or a weighted or unweighted average calculation method can be used to combine several different reliable depth values. Depth estimating device 416 may also update a confidence map of confidence values at this point.
    [0044] In step 770, the depth estimating device 416 advantageously joins the optimal depth values from the different level depth maps 618, 622, and 628 to generate the final optimized depth map. Finally, at step 774, the depth estimating device 416 can create a confidence map based on the confidence values corresponding to the optimal depth values of the final optimized depth map. The process in FIG. 7 can then end. The present invention, therefore, provides an improved system and method for generating a depth map using a multiple resolution procedure.
    [0045] Referring now to Fig. 8A, a fine-scale confidence characteristic diagram is shown, in accordance with an embodiment of the present invention. The embodiment of FIG. 8A is presented for purposes of illustration, and in alternative embodiments, the present invention may utilize reliability features other than those discussed in conjunction with the embodiment of FIG. 8A.
    [0046] In the embodiment of FIG. 8A, reliable features 812 include features of camera models/lens position 818 which may include any operational features or limitations of a particular camera 110 (FIG. 1). For example, 818 camera model/lens position features may include, but are not limited to, optimal features, lens position, zoom in/out position, camera calibration, etc. In one embodiment, features of camera models/lens position 818 may include a zoom in/out range that is supported by a given camera transmit data generation unit 110. In the embodiment of FIG. 8A, confidence features 812 further include implementation features 822 which may include any parameters or features of a particular algorithm that is used to generate a depth map. For example, implementation features 822 may include, but are not limited to, used block size and fit function used. In the embodiment of FIG. 8A , depth estimating device 416 can selectively combine confidence characteristics 812 in a weighted or unweighted manner to generate confidence values that represent a probability that corresponding depth values are accurate for fine-scale depth maps such as as a level 2 618 depth map (FIG. 6).
    [0047] Referring now to Fig. 8B, a coarse scale confidence characteristic diagram 814 is shown, in accordance with an embodiment of the present invention. The embodiment of FIG. 8B is presented for purposes of illustration, and in alternative embodiments, the present invention may utilize confidence features other than those discussed in conjunction with the embodiment of FIG. 8B.
    [0048] In the embodiment of FIG. 8B, coarse scale confidence feature 814 includes camera model/lens position features 818 and implementation features 822 that are similar to those identically named features discussed above in conjunction with FIG. 8A. In addition, coarse scale confidence characteristic 814 may include statistical characteristics 826 and measurement characteristics 830. In the embodiment of FIG. 8B, measurement characteristics 830 may include any characteristics that are based on appropriate measurement data. For example, measurement characteristics 830 may include, but are not limited to, motion vector doctors and pixel intensity measurements.
    [0049] In the embodiment of FIG. 8B, statistical features 826 may include any effective features derived from appropriate statistical analysis procedures. For example, the features of statistics 826 may include, but are not limited to, optimization rate statistics or a Sum of Absolute Differences (SAD) convergence technique which is further discussed below in conjunction with FIG. 8C. In the embodiment of FIG. 8B, depth estimating device 416 can selectively combine confidence characteristics 814 in a weighted or unweighted manner to generate confidence values that represent a probability that corresponding depth values are accurate for relatively coarse-scale depth maps such as such as the level depth map 0626 and the level depth map 1 622 (see FIG. 6).
    Referring now to Fig. 8C, a graph illustrating a Sum of Absolute Differences (SAD) convergence technique is shown, in accordance with an embodiment of the present invention. The embodiment of FIG. 8C is presented for purposes of illustration, and in alternative embodiments, the present invention can effect SAD convergence techniques with settings and parameters in addition to, or in place of, certain of those settings and parameters discussed in conjunction with the embodiment of FIG. 8C.
    [0051] The graph of FIG. 8C refers to statistical information derived from the depth estimation procedure using blurred images which is discussed above in conjunction with FIGS. 5B - 5C. In the graph of FIG. 8C, SAD values for out-of-focus blurred image pairs are shown on a horizontal axis, and iteration convergence velocities for a depth estimation procedure (see FIGS. 5B-5C) are shown on a vertical axis. The graph in FIG. 8C further includes a classifier curve 834 which can be derived in any effective way to indicate which depth values are reliable and which are unreliable. In certain embodiments, the classification curve 834 can be derived according to the following formulas.

    [0052] In certain modalities, the classification curve 834 can be generated based on empirical statistics of observed depth value data. In the embodiment of FIG. 8C, depth values that lie above classification curve 834 can be considered reliable, while depth values that lie below classification curve 834 can be considered unreliable.
    Referring now to Fig. 9, a diagram for one embodiment of a simplified confidence map 922 is shown, in accordance with the present invention. The confidence map of FIG. 9,922 is presented for purposes of illustration, and in alternative embodiments, confidence maps may include various other components and configurations in addition to, or in place of, those components and configurations discussed in conjunction with the embodiment of FIG. 9. For example, for purposes of simplicity, confidence map 922 of FIG. 9 is shown with only four confidence values. These four confidence values from confidence map 922 of FIG. 9 may correspond to the four depth values of the previous depth map 522 of FIG. 5A. However, confidence maps with any number of confidence values are also contemplated in the present invention.
    [0054] In the embodiment of FIG. 9, the trust map 922 can be generated in any effective way. For example, confidence map 922 can be generated during step 774 of Fig. 7C, as discussed above. In FIG. For example, confidence map 922 includes an A confidence value 926(a), a B confidence value 926(b), a C confidence value 926(c), and a D confidence value 926(d). In certain embodiments, confidence map 922 may be provided to various noise reduction modules that may choose to treat depth values with confidence values below a predetermined threshold as noise.
    [0055] The invention has been explained above with reference to certain embodiments. Other modalities will be apparent to those of skill in the art in light of this disclosure. For example, the present invention can readily be implemented using configurations and techniques other than those described in the embodiments above. Additionally, the present invention can effectively be used in conjunction with systems other than those described above. Accordingly, these and other variations on the discussed embodiments are intended to be covered by the present invention, which is limited only by the appended claims.
    权利要求:
    Claims (20)
    [0001]
    1. A system for generating a raw depth map, characterized in that it comprises: a memory (346) that includes: a depth map structure (614) that includes a plurality of depth map levels each of which has different characteristics of resolution; wherein the plurality of depth map levels include a fine-scale depth map (618), a medium-scale depth map (622), and a coarse-scale depth map (628); and, a depth estimating device (416) that evaluates depth values from the plurality of depth map levels to identify optimal depth values to populate an optimized depth map; wherein the depth estimating device (416) generates the fine-scale depth map (618) by applying a first compression ratio to an initial image data; and, a processor that executes instructions stored in memory (346).
    [0002]
    2. System according to claim 1, characterized by the fact that the depth estimation device (416) evaluates the depth values from the fine-scale depth map (618) using fine-scale confidence characteristics (812 ).
    [0003]
    3. System according to claim 2, characterized in that the fine-scale reliability characteristics (812) include camera model/lens position characteristics (818) and implementation characteristics (822), the characteristics of camera model/lens position (818) including optical characteristics, lens positions, zoom in/out positions, and camera calibration parameters, and implementation characteristics (822) including a block size and a function type of adjustment.
    [0004]
    4. System according to claim 3, characterized in that the depth estimation device (416) evaluates the depth values from the medium scale depth map (622) and the coarse scale depth map (628) using camera model/lens position characteristics (818), implementation characteristics (822), and coarse scale confidence characteristics (814).
    [0005]
    5. System according to claim 4, characterized in that the coarse scale (814) confidence characteristics include statistical characteristics (826) and measurement characteristics (830), the measurement characteristics (830) including measurements of motion vector and pixel intensity measurements, and statistical features (826) including optimization rate statistics and a Sum of Absolute Differences convergence technique.
    [0006]
    6. System according to claim 5, characterized by the fact that the Sum of Absolute Differences convergence technique uses a classification curve (834) to indicate which of the depth values are reliable and which are unreliable, the classification curve (834) being generated based on empirical statistics of the observed depth value data.
    [0007]
    7. System according to claim 1, characterized in that the first compression ratio is 1/4, the fine-scale depth map (618) having a configuration of 4 X 4 blocks.
    [0008]
    8. System according to claim 1, characterized in that the depth estimation device (416) evaluates the fine-scale depth map (618) with selected fine-scale confidence characteristics (812).
    [0009]
    9. System according to claim 8, characterized in that the depth estimation device (416) categorizes depth values from the fine-scale depth map (618) as one of low confidence depth values , medium confidence depth values, and high confidence depth values.
    [0010]
    10. System according to claim 1, characterized in that the depth estimation device (416) generates the medium-scale depth map (622) by applying a second compression ratio to the initial image data, the depth estimating device (416) also generating the coarse scale depth map (628) by applying a third compression ratio to the initial image data.
    [0011]
    11. System according to claim 10, characterized in that the second compression ratio is 1/8, the medium-scale depth map (622) having a configuration of 2 X 2 blocks, the third compression ratio being 1/16, the coarse scale depth map (628) having a single block configuration.
    [0012]
    12. System according to claim 10, characterized in that the depth estimation device (416) evaluates the medium scale depth map (622) and the coarse scale depth map (628) with characteristics of coarse scale confidence (814) selected to determine any reliable depth value candidates.
    [0013]
    13. System according to claim 12, characterized in that the depth estimation device (416) performs a spatial resizing procedure to adjust the block size of the coarse scale depth map (628) and the map from medium scale depth (622) to fine scale depth map block size (618).
    [0014]
    14. System according to claim 13, characterized in that the depth estimation device (416) performs a magnitude sizing procedure to adjust depth value intervals of the coarse scale depth map (628) and from medium scale depth map (622) to depth value range of fine scale depth map (618).
    [0015]
    15. System according to claim 14, characterized in that the depth estimation device (416) performs an update procedure to update the optimal depth values with reliable depth value candidates using one of a technique selection and an averaging technique.
    [0016]
    16. System according to claim 15, characterized in that the depth estimation device (416) performs a joining procedure to create the optimized depth map from the optimal depth values of the scaled depth map fine (618), the medium-scale depth map (622), and the coarse-scale depth map (628).
    [0017]
    17. System according to claim 16, characterized in that the depth estimation device (416) generates a final confidence map corresponding to the optimized depth map.
    [0018]
    18. Method for generating a raw depth map, characterized in that it comprises: in a device comprising a processor: creating a depth map structure (614) that includes a plurality of depth map levels which each have different resolution characteristics; wherein the plurality of depth map levels include a fine-scale depth map (618), a medium-scale depth map (622), and a coarse-scale depth map (628); and using a depth estimating device (416) to evaluate depth values from the plurality of depth map levels to identify optimal depth values to populate an optimized depth map; wherein the depth estimating device (416) generates the fine-scale depth map (618) by applying a first compression ratio to an initial image data.
    [0019]
    19. A system for generating a raw depth map, characterized in that it comprises: a memory (346) that includes: a depth map structure (614) that includes a plurality of depth map levels in which each has different resolution characteristics; wherein the plurality of depth map levels include a fine-scale depth map (618), a medium-scale depth map (622), and a coarse-scale depth map (628); and, a depth estimating device (416) that evaluates depth values from the plurality of depth map levels to identify optimal depth values to populate an optimized depth map; wherein the depth estimating device (416) evaluates the depth values from the medium scale depth map (622) and the coarse scale depth map (628) using the camera model/lens position characteristics (818), implementation characteristics (822), and coarse scale confidence characteristics (814); coarse scale (814) confidence characteristics include statistical characteristics (826) and measurement characteristics (830); the measurement characteristics (830) including motion vector measurements and pitch intensity measurements; and, a Sum of Absolute Differences convergence technique; and, a processor that executes instructions stored in memory (346).
    [0020]
    20. System according to claim 19, characterized by the fact that the Sum of Absolute Differences convergence technique uses a classification curve (834) to indicate which of the depth values are reliable and which are unreliable, the classification curve (834) being generated based on empirical statistics of the observed depth value data.
    类似技术:
    公开号 | 公开日 | 专利标题
    BR102012030034B1|2021-05-25|system and method for generating a raw depth map
    JP6731738B2|2020-07-29|Image generation apparatus for generating depth map using phase detection pixels
    US10269130B2|2019-04-23|Methods and apparatus for control of light field capture object distance adjustment range via adjusting bending degree of sensor imaging zone
    JP5980294B2|2016-08-31|Data processing apparatus, imaging apparatus, and data processing method
    US9092875B2|2015-07-28|Motion estimation apparatus, depth estimation apparatus, and motion estimation method
    JP6735592B2|2020-08-05|Image processing apparatus, control method thereof, and image processing system
    JP5961945B2|2016-08-03|Image processing apparatus, projector and projector system having the image processing apparatus, image processing method, program thereof, and recording medium recording the program
    US10475237B2|2019-11-12|Image processing apparatus and control method thereof
    JP2014057303A|2014-03-27|System and method for utilizing enhanced scene detection in depth estimation procedure
    BR102015004240A2|2016-03-15|image processing apparatus and method for controlling the same
    Falquez et al.2016|Inertial aided dense & semi-dense methods for robust direct visual odometry
    TW201337838A|2013-09-16|System and method for performing depth estimation utilizing defocused pillbox images
    US9020280B2|2015-04-28|System and method for evaluating focus direction under various lighting conditions
    JP2017220885A|2017-12-14|Image processing system, control method, and control program
    Orozco et al.2012|Full high-dynamic range images for dynamic scenes
    JP5887974B2|2016-03-16|Similar image region search device, similar image region search method, and similar image region search program
    US10154176B1|2018-12-11|Calibrating depth cameras using natural objects with expected shapes
    JP2019037004A|2019-03-07|Image processing apparatus, control method thereof, and program
    JP2019106215A|2019-06-27|Data processor, imaging device, and data processing method
    KR20190042472A|2019-04-24|Method and apparatus for estimating plenoptic camera array depth images with neural network
    JP2020201823A|2020-12-17|Image processing device, image processing method, and program
    JP2016194956A|2016-11-17|Data processor, imaging device, and data processing method
    JP6525693B2|2019-06-05|Image processing apparatus and image processing method
    JP2018191261A|2018-11-29|Image processing apparatus, imaging apparatus, image processing method, program, and storage medium
    WO2018161322A1|2018-09-13|Depth-based image processing method, processing device and electronic device
    同族专利:
    公开号 | 公开日
    US9161010B2|2015-10-13|
    CN103139469B|2016-05-18|
    BR102012030034A2|2013-11-12|
    KR101364421B1|2014-02-17|
    JP2013117969A|2013-06-13|
    CN103139469A|2013-06-05|
    EP2600618A2|2013-06-05|
    EP2600618A3|2015-04-01|
    US20130142415A1|2013-06-06|
    KR20130061636A|2013-06-11|
    TWI519128B|2016-01-21|
    JP5454821B2|2014-03-26|
    TW201338500A|2013-09-16|
    引用文献:
    公开号 | 申请日 | 公开日 | 申请人 | 专利标题
    JPH06176107A|1992-12-04|1994-06-24|Canon Inc|Method and device for processing distance picture|
    JPH1051675A|1996-07-29|1998-02-20|Ikegami Tsushinki Co Ltd|Focus controller, its method and storage medium|
    JP4517449B2|2000-05-10|2010-08-04|株式会社豊田中央研究所|Correlation calculation method for images|
    US20030235338A1|2002-06-19|2003-12-25|Meetrix Corporation|Transmission of independently compressed video objects over internet protocol|
    JP4069468B2|2004-01-15|2008-04-02|充夫 松田|Image forming device|
    JP4792929B2|2005-11-14|2011-10-12|株式会社ニコン|Digital camera|
    US8970680B2|2006-08-01|2015-03-03|Qualcomm Incorporated|Real-time capturing and generating stereo images and videos with a monoscopic low power mobile device|
    DE102006055641B4|2006-11-22|2013-01-31|Visumotion Gmbh|Arrangement and method for recording and reproducing images of a scene and / or an object|
    JP2008309637A|2007-06-14|2008-12-25|Konica Minolta Holdings Inc|Obstruction measuring method, obstruction measuring apparatus, and obstruction measuring system|
    IL191615A|2007-10-23|2015-05-31|Israel Aerospace Ind Ltd|Method and system for producing tie points for use in stereo-matching of stereoscopic images and method for detecting differences in a photographed scenery between two time points|
    US8233077B2|2007-12-27|2012-07-31|Qualcomm Incorporated|Method and apparatus with depth map generation|
    JP5062531B2|2008-06-03|2012-10-31|コニカミノルタホールディングス株式会社|Image processing apparatus and image processing method|
    EP2184713A1|2008-11-04|2010-05-12|Koninklijke Philips Electronics N.V.|Method and device for generating a depth map|
    US8164617B2|2009-03-25|2012-04-24|Cisco Technology, Inc.|Combining views of a plurality of cameras for a video conferencing endpoint with a display wall|
    US8643701B2|2009-11-18|2014-02-04|University Of Illinois At Urbana-Champaign|System for executing 3D propagation for depth image-based rendering|
    US8027582B2|2009-12-21|2011-09-27|Sony Corporation|Autofocus with confidence measure|
    US8045046B1|2010-04-13|2011-10-25|Sony Corporation|Four-dimensional polynomial model for depth estimation based on two-picture matching|
    WO2011138472A1|2010-05-07|2011-11-10|Telefonica, S.A.|Method for generating depth maps for converting moving 2d images to 3d|JP2013172190A|2012-02-17|2013-09-02|Sony Corp|Image processing device and image processing method and program|
    JP6140935B2|2012-05-17|2017-06-07|キヤノン株式会社|Image processing apparatus, image processing method, image processing program, and imaging apparatus|
    US9240048B2|2012-11-30|2016-01-19|Adobe Systems Incorporated|Depth estimation using three-dimensional epipolar data structures|
    US9214025B2|2012-11-30|2015-12-15|Adobe Systems Incorporated|Depth estimation using normalized displacement of image pairs|
    WO2015046183A1|2013-09-24|2015-04-02|味の素株式会社|Glycoamino acid and use thereof|
    US9418432B2|2014-03-28|2016-08-16|Sony Corporation|Imaging system with depth estimation mechanism and method of operation thereof|
    KR101882931B1|2014-07-10|2018-07-30|삼성전자주식회사|Multi view image display apparatus and disparity estimation method of thereof|
    US9292926B1|2014-11-24|2016-03-22|Adobe Systems Incorporated|Depth map generation|
    CN105721852B|2014-11-24|2018-12-14|奥多比公司|For determining the method, storage equipment and system of the capture instruction of depth refined image|
    US10404969B2|2015-01-20|2019-09-03|Qualcomm Incorporated|Method and apparatus for multiple technology depth map acquisition and fusion|
    EP3391647A4|2015-12-18|2019-12-11|Boe Technology Group Co. Ltd.|Method, apparatus, and non-transitory computer readable medium for generating depth maps|
    US11272163B2|2017-02-07|2022-03-08|Sony Corporation|Image processing apparatus and image processing method|
    CN108810317B|2017-05-05|2021-03-09|展讯通信（上海）有限公司|True motion estimation method and device, computer readable storage medium and terminal|
    CN108234858B|2017-05-19|2020-05-01|深圳市商汤科技有限公司|Image blurring processing method and device, storage medium and electronic equipment|
    WO2019061064A1|2017-09-27|2019-04-04|深圳市大疆创新科技有限公司|Image processing method and device|
    CN109410261B|2018-10-08|2021-12-10|浙江科技学院|Monocular image depth estimation method based on pyramid pooling module|
    CN112102386A|2019-01-22|2020-12-18|Oppo广东移动通信有限公司|Image processing method, image processing device, electronic equipment and computer readable storage medium|
    法律状态:
    2013-11-12| B03A| Publication of a patent application or of a certificate of addition of invention [chapter 3.1 patent gazette]|
    2018-12-04| B06F| Objections, documents and/or translations needed after an examination request according [chapter 6.6 patent gazette]|
    2019-10-15| B06U| Preliminary requirement: requests with searches performed by other patent offices: procedure suspended [chapter 6.21 patent gazette]|
    2021-03-30| B09A| Decision: intention to grant [chapter 9.1 patent gazette]|
    2021-05-25| B16A| Patent or certificate of addition of invention granted [chapter 16.1 patent gazette]|Free format text: PRAZO DE VALIDADE: 20 (VINTE) ANOS CONTADOS A PARTIR DE 26/11/2012, OBSERVADAS AS CONDICOES LEGAIS. |
    优先权:
    申请号 | 申请日 | 专利标题
    US201161565839P| true| 2011-12-01|2011-12-01|
    US61/565,839|2011-12-01|
    US13/430,915|2012-03-27|
    US13/430,915|US9161010B2|2011-12-01|2012-03-27|System and method for generating robust depth maps utilizing a multi-resolution procedure|
    [返回顶部]